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ABSTRACT 

The purpose of this study was to determine whether it 
is both possible and cost-effective to revise middle-difficulty 
Graduate Record Examinations (GRE) discrete items in order to produce 
items. of higher or lower difficulty. The. basic procedure was to 
select items of a given difficulty and, by revising the distractors, 
make them easier or more difficult. It was found that it is 
significantly easier to increase the difficulty of middle-difficulty 
items than to reduce the difficulty of such items and that the 
difficulties of antonyms and analogies are much easier to manipulate 
than those of sentence completions. The evidence also suggests that 
producing harder analogies and antonyms by revising items in this 
manner would be a cost-effective procedure. Nine appendixes present 
definitions of GRE content categories, examples of revised items, 
statistical analysis results for revised items and the tests, pretest 
yields, and cost figures. Three tables illustrate the discussion. 
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Abstract 



The purpose of this study was to determine whether it is both possible and cost-effective 
to revise middle-difficulty GRE discrete items in order to produce items of higher or lower 
difficulty. It was found that it is significantly easier to increase the difficulty of middle-difficulty 
items than to reduce the difficulty of such items, and that the lifficulties of antonyms and 
analogies are much easier to manipulate than those of sentence completions. The evidence also 
suggests that producing harder analogies and antonyms by revising items in this manner would be 
a cost-effective procedure. 



Introduction 



Over the past several years, assemblers of the verbal section of the GRE General Test 
have experienced increasing problems in meeting the current standard deviation of the delta 
specification, that is, meeting the statistical specification that calls for a relatively wide range of 
both hard and easy items in a verbal final form. These problems are the result of two somewhat 
related factors: a continuing inability to obtain, predictably and consistently, discrete items 
(analogies, antonyms, and sentence completions) that are very easy (A 6-8), arid a similar, though 
even more serious, inability to obtain difficult discrete items (A 14-17).' In addition, the problems 
of assembly have been further complicated by losses of discrete items caused by the differential 
item functioning (DIF) statistic, a statistical screening for items that function differentially for 
subgroups of examinees; these losses, too, seem greatest at the extremes of the delta scale. The 
situation has, in fact, reached the point that assembly of the October 1989 final forms was delayed 
in order to wait for high delta items from the October 1988 pretests, because there was an 
insufficient pool of hard and easy itenis available for assembly. It is apparent that, even though 
modification of the standard deviation specification is under study and that statistical specifications 
based on item response theory (IRT) may relieve the problem to some degree in the future, the 
need to obtain items of a targeted difficulty, especially high delta items, is now critical and will 
remain so for the immediate future. The purpose of this study, therefore, is to address the 
continuing need for items with particular statistical characteristics (difficulty level) in the verbal 
measure. The study includes attempts at both raising and lowering difficulty levels of middle 
difficulty items (generating hard and easy items). 



Method 

A large pool of middle-difficulty, pretested discrete items already exists and, under current 
pretesting conditions, can never be exhausted in assembling GRE final forms. This pool of items 
provided the material for the study. The basic procedure was to select items of difficulty A 9-10 
and, by revising the distractors, make such items easier; similarly, items of difficulty A 12-13 were 
revised in an effort to make them more difficult. Items were selected as follows: 



'Delta is an index of item difficulty based upon the percent of all candidates trying the item who 
answered it correctly. The principle advantage of delta over P+ (percent correct) lies in the fact that equal 
increments in delta, unlike P+, may reasonably be assumed to represent equal increments in difficulty. 
This characteristic linearity of delta permits comparisons to be made between groups taking different test 
forms. A theoretical average delta for a 5-choice item is 12.0. (If 50% of the candidates know the answer 
and the other 50% answer by chance, 60%; of the students will answer the item correctly. A P+ of .60 
corresponds to a delta of 12.0) Deltas ordinarily range from 6.0 for very easy items (approximately 95% 
correct) to 13.0 for middle difficulty items (approximately 50% correct) to 20.0 for a very hard item 
(approximately 5% correct). The average range for the GRE verbal measure is 6.5 to 16.5, with a mean 
for the test of 12.0. 
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Content 


Delta 


Number 


Intended 


Type 


Qassification^ 


Range 


of Items 


Outcome 


Sentence 


Art-Humanities 


9-10 


5 


lower delta 


Completions 


Art-Humanities 


12-13 


10 


higher delta 


Human Rels 


9-10 


5 


lower delta 




Human Rels 


12-13 


10 


higher delta 


Antonyms 


Art-Humanities 


9-10 


5 


lower delta 


Art-Humanities 


12-13 


10 


higher delta 




Human Rels 


9-iO 


5 


lower delta 




Human Rels 


12-13 


10 


higher delta 


Analogies 


Art-Humanities 


9-10 


5 


lower delta 


Art-Humanities 


12-13 


10 


higher delta 




Human Rels 


9-10 


5 


lower delta 




Human Rels 


12-13 


10 


higher delta 



Item Characteristics 

In the case of attempts to reduce the difficulty level of verbal discretes, items were 
selected that had at least two strong distractors, that is, distractors that were attracting a 
significant number of test-takers. Distractors were weakened, in general, by relying on the 
judgments of experienced item writers to lower the vocabulary level and reduce the closeness of 
the distractor to the credited option. Specifically, for antonyms, words were chosen as distractors 
that were less semantically appropriate for either the stem or answer contexts; for sentence 
completion items, words were chosen as distractors that were less likely tc appear in the contexts 
of the given sentences; and for analogy distractors, word pairs were chosen whose analogical 
relationships were less similar to those of the stem-answer pairs. (See Appendix B for examples 
of the revisions.) 

For increasing the difficulty level of verbal discretes, items were selected that had at least 
two weak distractors, that is, distractors that were attracting relatively few test takers. 
Strengthening a distractor, in general, consisted of relying on the judgment of experienced item 
writers to raise the vocabulary level and increase the closeness of the distractor to ihe credited 
option. For antonyms, words were chosen as distractors that were more semantically appropriate 
for either stem or answer contexts; for sentence completion items, words were chosen as 



■These classifications have been selected as those that experience shows are least likely to yiclU high 
DIF values. See Appendix A for the definitions of the entire set of content categories for GRE verbal 
discretes. 
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distractors that were more likely to appear in the contexts of the sentences; and for analogy 
distractors, pairs of words were chosen whose analogical relationships were more similar to those 
of the stem-answer pairs. (See Appendix C for examples of the revisions.) 

The revised items were assembled into three experimental sections having the following 
characteristics: 

(1) Each section contained a long and short reading comprehension set typical of 
GRE verbal pretests and final forms; that is, each section maintained the 
appearance of a standard GRE pretest/final form. 

(2) Each section contained both easy and difficult discrete items, although not in the 
same proportions as in GRE verbal pretests and final forms. 

(3) Each section contained several science and practical affairs items in addition to 
the human relations and arts and humanities items being studied, again to 
preserve the appearance of the experimental sections as standard verbal pretests. 

Results 

The complete quantitative results of this study can be found in Appendix D and Appendix 
E. Table 1 presents a summary of the most significant data, based on an analysis of those items 
(67 out of 72, or 93%) that had acceptable correlation coefficients (ry, > .20) after pretesting. All 
had acceptable r,,!, before revision. 

Our general finding is that manipulating the difficulty of a verbal discrete item in a desired 
direction by changing some of the distractors is possible. The ability to manipulate the difficulty, 
however, depends on a number of factors. First, as Table 1 shows, the average reductions in delta 
achieved during this study (-0.33) were smaller than the average increases in delta that were 
achieved (-f-0.78). Second, the difficulty levels of sentence completions (SNCP) proved 
significantly less manipulable than those of antonyms (ANT) or especially analogies (ANAL). 
Looking only at those items that were intended to be made harder, it can be seen in Table 1 that, 
whereas the difficulty levels of sentence completions were raised on average by only -f-0.28 delta 
points, the difficulty levels of antonyms and analogies together showed an average increase of 
4-0.97 delta points. In addition, the standard deviations of the deltas of analogies and antonyms 
indicate that about 66% of all tries yielded increases of between 0 and 2 delta points. 

We also compared the cost-effectiveness of the experimental method for producing 
difficult discretes with the current method of pretesting. Two general methods of assessing the 
cost/benefits were used. First, we attempted to compare the dollar value ($value) of the items 
produced by each method with the cost of production for each method. We assumed first that" 
the experimental method would be used only on antonyms and analogies and only to increase 
deltas. We then used the frequency data given in Appendix F to determine what distribution of 
deltas, on average, could be expected from a set of 20 analogies and antonyms (one pretest's 
worth), revised from items with difficulties between Al2 and Al3 and having a mean delta of 12.6, 
and compared that to the standard yields of a normal pretest (the data from which the normal 
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Table 1 








Statistical Results For Acceptable Items (N=67) 






Item 


Avg Change 






Objective 


Type 


in Delta 


Range 


StdDcv 


Make Easier 


SNCP 


-0.26 


-1.4 to 1.5 


0.92 




ANAL 


-0.53 


-2.1 to 1.1 


1 (VI 




ANT 


-0.25 


-2.0 to 2.0 


1 AA 




ANAL + ANT 


-0.38 








AVG 


-0.33 




■ 


Make Harder 


SNCP 


+0.28 


-2.9 to 1.6 


1.39 




ANAL 


+ 1.10 


+0.2 to 3.5 


1.07 




ANT 


+0.88 


-1.0 to 3.1 


0.99 




ANAL + ANT 


+0.97 








AVG 


+0.78 






Control {N= 


26) 


+0.12 






Distribution of Items Whose Statistical Change Was in Intendccl Direction 


AChange 


0.3-0.7 0.8-1.2 


1.3-1.7 1.8-2.2 


2.3-2.7 2.8-3.2 


3.3-3.7 


# of items 


8 15 


9 5 


0 3 


1 = 41 



and compared that to the standard yields of a normal pretest (the data from which the normal 
yields have been calculated are contained in Appendix G). Table 2 compares the two sets of 
yields. 

As Table 2 demonstrates, the experimental method produces difficult items at significantly 
higher rates than those produced by standard pretesting. Using the calculation of the $values of 
GRE items by delta range found in Appendix H, the $values of the items produced by standard 
pretesting and by the experimental procedure can be compared (see Table 3). These values show 
that for one pretest's worth of analogies and antonyms (N=20), the experimental procedure 
produces items of 42% greater value than those produced by the standard pretesting procedure. 
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Table 2 



Distribution of Deltas for 20 Hard Analogies and Antonyms: 
Standard versus Experimental 



Delta 



Standard 
Frequency 
(Good R«.) 



Experimental 
Frequen(^ 
(GoodRtj.) 



12 
13 
14 
15 
>16 



2.8 
2.2 
2.0 
1.1 
0.5 



5.7 
6.4 
2.9 
2.1 
0.7 



A comparison of the costs of production for each of these methods, however, yields 
unclear evidence about the cost-effectiveness of the experimental method. Because the 
experimental items can be produced more quickly than items produced through standard 
pretesting (1.2 hours/exp.item vs. 1.8 hours/std.item), test development costs for pretesting 20 
analogies and antonyms are $850 less for the experimental method than the costs of the standard 
method ($7,660 exp. cost vs. $8,550 std. cost). However, before a cost/benefits ratio can be 
calculated, the value of the pretested items used must be factored into the costs. For standard 
pretesting, this value is $0 (because there are no pretested items), yielding a cost/benefits ratio of 
$8,550:$ 13,457 or $1:$1.57. (See Table 3.) 

For the experimental items, initial value can be calculated in a number of ways. Given the 
large GRE pool of Al2 analogies and antonyms and the fact that in a standard year of 20 pretests 
and 4 final forms 100% more Al2s and 24% more Al3s will be produced than will be needed, one 
reasonable assessment of the initial value of the discretes used in the experimental method is also 
$0. This results in a costA)enefits ratio of $7,660:$ 19, 173 or $1:$2.50. Using the determination of 
values of GRE discretes of various deltas listed in Appendix H, however, results in an initial value 
for the items used in the experimental method of $10,985 (14 Al2s, and 6 Al3s), and a 
cost/benefits ratio of $18,645:$ 19,173 or $1:$1.03. Thus, using the first estimation of initial value, 
the experimental method produces approximately one-half times more benefit per dollar spent 
than does the standard method. By the second estimation of initial value, however, the 
experimental method produces only about the same benefit per dollar spent as does the standard 
method. We believe that estimating the initial value of the revised items at $0 is the more 
reasonable way to proceed, because large portions of the GRE pool of middlc-dilTicully items will, 
in all probability, never be used. 

A second way to calculate the cost/benefits of the experimental method is to determine 
how many fewer pretests could be run if a mixture of standard method pretests and experimental 
method pretests were used. Using the data found in Appendix I, it was determined that a mixture 
of 14 standard pretests and 3 experimental pretests would best yield the requisite number of items 



ERIC 



6 



Table 3 

Value Produced in Pretesting 20 Analogies and Antonyms: 
Standard versus Eiqierimcntal Method 
(Items with Good Ry.) 





SValuc/ 


Std 


Exp 


Std 


Exp 


Delta 


Delta* 


Freq** 


Freq*** 


$Value 


SValue 


<8 


757 


0.6 


0.0 


482 


0 


8 


757 


1.1 


0.0 


826 


0 


9 


892 


1.8 


0.0 


1,621 


0 


10 


245 


2.0 


0.0 


491 


0 


11 


203 


3.3 


0.7 


665 


145 


12 


403 


2.8 


5.7 


1,134 


2,300 


13 


892 


2.2 


6.4 


1,945 


5,731 


14 


757 


2.0 


2.9 


1,515 


2,164 


15 


3,092 


1.1 


2.1 


3,373 


6,625 


>16 


3,092 


0.5 


0.7 


1,405 


2,208 


TOTAL 








$13,457 


$19,173 



*See Appendix H. 



**Sce Appendix G. 



*See Appendix F. 



at the various deltas where they ore needed.' If the value of the pret.-sted items used in the 
experimental pretests is set at $0, ihe savings per year would be $62,550 ($20,000 [T.D. 
costs/pretest] x 3 [pretests saved/yi) + $850 [T.D. savings/exp.pretest] x 3 [exp. pretests/yr]). If 
the values listed in Appendix H are used, the savings per year would be $29,596 ($62,550 - 
$10,985 [value of items/exp. pretest] x 3 [exp. pretcsts/yr]). 

The preceding shows that a consistent estimate of the savings that could be achieved by 
including the experimental method in GRE pretesting is difficult to come by, mainly because 
there is no one obvious way to determine the value of the already pretested items reused in the 
experimental pretests. Our best guess, using the figures for pretest savings, is that a mixture of 
standard and experimental pretests could save the GRE programs between $30,000 and $60,000 
per year if we can extrapolate from the data on which our study is based, and if we can safely cut 



•'Because the experimental items were written in content categories chosen to minimize DIF, it is 
possible that the yields listed in Appendix 1 for the experimental pretests might in practice be lower. 
However, because the pool of pretested items from which the experimental items were picked was 
deliberately restricted to older items for which no DIF data were available, it is likely that these 
experimental items show more DIF than would occur if revision was made of items for which DIF 
information was already known. 
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back to 17 pretests per year. 



Conclusions 

The results of our research indicate thdt the experimental procedure can be used to enrich 
the GRE discrete pool, but only with the following provisos: (1) The method should be used only 
to produce higher delta items; and (2) the method should not be applied to sentence completion 
items. We believe that this method is likely to be a cost-effective way to produce more difficult 
analogy and antonym discretes, and that it can be effectively introduced by dedicating three 
pretests worth of analogies and antonyms per year to items revised in the wr.ys suggested in this 
study. 

We also believe that further research is in order before the benefits of this method are 
certain. We recommend (1) trying a more extensive study using all four content categories and 
focusing specifically on increasing the difficulty of antonyms and analogies; (2) studying the ability 
to increase the difficulty of discretes that are not middle range in difficulty (i.e., is it possible by 
using the same techniques to raise Al4 and Al5 items to Al5, Al6, and Al7 items?); (3) doing a 
more extensive investigation of the possibility of making AlO and All analogies easier using this 
method; and (4) using this method of research to determine, for each item type, the degree to 
which distractors affect item performance (i.e., it seems from data on SNCP items that 
performance on those items is not much affected by the distractors). 
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APPENDIX A: DEFINITIONS OF ORE CONTENT CATEGORIES 



Art-Humanities 

Thiis category includes words (or, in the case of sentence completions, issues and ideas) 
that are most likely to be used or discussed in the discourse of, or discourse about, the 
fine and applied arts (painting and architecture, for example), literature, philosophy, 
religion, and other such fields. 

Social Studies and Practical or Everyday Life 

This category includes words (or, in the cases of sentence completions, issues and ideas) 
that are most likely to be used or discussed in the discourse of, or discourse about, such 
fields as communications, business, politics and government, economics, and 
transportation. 

Science and Nature 

This category includes words (or, in the case of sentence completions, issues and ideas) 
that arc most likely to be used or discussed in the discourse of, or discourse about, such 
fields as biology, chemistry, physics, geology, and the theoretical and applied sciences, such 
as mathematics and medicine. 



Human Relationships and Feelings 

This category includes words (or, in the case of sentence completions, issues and ideas) 
that concern emotions, interpersonal relationships, or analyses of character. 
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APPENDIX B: EXAMPLES OF ITEMS REVISED TO BECXDME EASIER 



ITEM: V-081807 

Original Item 

STiAJsrZA:POEM:: 

(A) play:drama 

(B) lyflc:song 

(C) cbapter:book 

(D) stone:statue 

(E) reproduction:painting 



BASE N 

1,995 


OMIT 

91 


A 

21 


B 

349 


c * 
1511 


D 

16 


E 

5 


M-TOT 
13.0 


SCALE 

NGRl 


AE 
9.4 


CRIT 

XS80 


ITEM # 

14 


M-0 

8.3 


M-A 

6.5 


M-B 
11.2 


MrC 
13.9 


M-D 

8.6 


M-E 
8.0 


P-TOT 
1.00 


P+ 

0.76 


AO 
10.2 


Rbis 
0.52 



Revised Item 
STANZA:POEM:: 

(A) play.drama 

(B) miisic:soiig 

(C) chapter:ibook 

(D) stone:statue 

(E) reproduction:painting 



BASE N 

1,045 


OMIT 

0 


A 

10 


B 

18 


c * 
1000 


D 

15 


E 

2 


M-TOT 
13.0 


SCALE 

3DGR 


AE 
7.3 


CRIT 

XS76 


ITEM # 

8 


M-0 

0.0 


M-A 

6.5 


M-B 

7.7 


M-C 

13.2 


M-D 

9.7 


M-E 
11.0 


P-TOT 
1.00 


P+ 

0.96 


AO 


Rbis 

0.53 



* indicates key 
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Appc . ^\x B, continued 10 
ITEM: V-096399 

Original Item 

RESENT: 

(A) accommodate 

(B) welcome 

(C) protect 

(D) coixcem 

(E) insist 



BASE N 


OMIT 


A 


B * 


C 


D 


E 


M-TOT 


SCALE 


AE 


CRIT 


1,470 


4 


154 


1098 


55 


80 


42 


13.1 


3DGR 


9.8 


XS76 


ITEM # 


H-0 


M-A 


M-B 


M-C 


M-D 


M-E 


p. TOT 


P+ 


AO 


Rbis 


30 


10.0 


12.1 


13.6 


10.7 


11.0 


10.2 


0.97 


0.77 


10.1 


0.33 



Revised Item 

RESENT: 

(A) Hasten 

(B) welcome 

(C) protect 

(D) enipldy 

(E) demand 



BASE N 

1,045 


OMIT 

1 


A 

10 


B * 

970 


C 

12 


D 

42 


E 

9 


M-TOT 
13.0 


SCALE 

3DGR 


AE 
8.1 


CRIT 

XS76 


ITEM # 

28 


M-0 
10.0 


M-A 

7.8 


M-B 

13.2 


M-C 
10.3 


M-D 

10.4 


M-E 
9.9 


p. TOT 
1.00 


P+ 

0.93 


AO 
7.1 


Rbis 
0.40 
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ITEM: V-fl81813 

Original Item 

Even though most dance critics believe that the choreographer succeeded in the 

public's notions of how dance should be performed, her radical ideas have in fact not yet 
been fully . 

(A) <a3it{riuiictiftg..revealed 

(B) challenging..negated 

(C) reshaping, .accepted 

(D) c8pturing.,developi?d 

(E) cqnftfiaiftgwitBdetstood 



BASE N 

1,995 


OMIT 

40 


A 

44 


B 

41 


C * 
1468 


D 

227 


E 

173 


M-TOT 
13.0 


SCALE 
NGRl 


AE 
9.7 


GRIT 
XS80 


ITEM # 

19 


M-0 

8.5 


M-A 

11.4 


M-B 

9.9 


M-C 
13.8 


M-D 
10.9 


M-E 
11.3 


P-TOT 
1.00 


P+ 

0.74 


AO 
10.4 


Rbis 
0.45 



Revised Item 

Even though most dance critics believe that the choreographer succeeded in the 

public's notions of how dance should be performed, her radical ideas have in fact not yet 
been fully 

(A) pre«eatin^..arr8iiged 

(B) chailenging..negated 

(C) reshaping..accepted 

(D) contfolHttg..e}cercised 

(E) underestitnatingMreguJatcd 



BASE N 

1,045 


OMIT 

1 


A 

32 


B 

30 


C * 
962 


D 

18 


E 

2 


M-TOT 

13.0 


SCALE 

3DGR 


AE 
8.3 


CRIT 

XS76 


ITEM # 

1 


M-0 

12.0 


M-A 

10.4 


M-B 

9.6 


M-C 
13.3 


M-D 

8.7 


M-E 

8.5 


P-TOT 
1.00 


P+ 

0.92 


AO 
7.4 


Rbis 
0.44 
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APPENDIX C: EXAMPLES OF ITEMS REVISED TO BECOME HARDER 



ITEM: V-083094 

Ori ginal Item 

LOQUACIOUS:CHATrER:: 

(A) perHo«s:safeguard 

(B) numerousxount 

(C) marvelousjadmite 

(D) officious:meddle 

(E) preciousxherish 



BASE N 

1.515 


OMIT 

138 


A 

115 


B 

235 


C 

183 


D * 

590 


E 

231 


M-TOT 

13.1 


SCALE 

NGRl 


AE 
12.6 


CRIT 

XS75 


ITEM # 

47 


M-0 

10.7 


M-A 

11.8 


M-B 

12.4 


M-C 

11.1 


M-D 

15.0 


M-E 

12.4 


P-TOT 
0.98 


P+ 

0.40 


AO 
14.1 


Rbis 
0.50 



Revised Item 

LOQUACIOUS:CHATTER:: 

(A) meticuloustcomplain 

(B) numerous:count 

(C) vbraciomstarve 

(D) officious:meddie 

(E) preciousxherish 



BASE N 

1.015 


OMIT 

3 


A 

116 


B 

107 


C 

224 


D * 
446 


E 

119 


M-TOT 
13.0 


SCALE 

3DGR 


AE 
13.6 


CRIT 

XS76 


ITEM # 

16 


M-0 

9.3 


M-A 

10.6 


M-B 
11.6 


M-C 

13.0 


M-D 

14.3 


M-E 

11.8 


P-TOT 
1.00 


P+ 

0.44 


AO 
13.6 


Rbis 
0.37 



ERIC 
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Appendix C, continued 13 
ITEM: V-079231 

Original Item 

DUBITABLE: 

(A) essential . 

(B) certain 

(C) seasible 

(D) decisive 

(E) verifiable 



BASE N 

795 


OMIT 

262 


A 

20 


B * 

244 


C 

29 


D 

44 


E 

101 


M-TOT 

13.3 


SCALE 

NGRl 


AE 

12.8. 


CRIT 
XS80 


ITEM # 

50 


M-0 

11.7 


M-A 

11.8 


M-B 

15.1 


M-C 

10.1 


M-D 

11.5 


M-E 

14.9 


p. TOT 

0.88 


P+ 

0.35 


AO 
14.8 


Rbis 
0.44 



Revised Item 
DUBITABLE: 

(A) reassuring 

(B) verifiable 

(C) coercive 

(D) decisive 

(E) certain 



BASE N 

1,015 


OMIT 

6 


A 

127 


B 

250 


C 

150 


D 

99 


E * 

357 


M-TOT 
13.1 


SCALE 

3DGR 


AE 
14.3 


CRIT 

XS76 


ITEM # 

36 


M-0 

12.7 


M-A 

12.4 


M-B 

13.3 


M-C 

10.3 


M-D 

11.0 


M-E 

14.8 


p. TOT 

0.97 


P+ 

0.36 


AO 
14.4 


Rbis 
0.43 



1.8 



ERIC 
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ITEM: V-099427 

Original Item 

In the midst of so many evasive comments, this forthright statement, whatever its intrinsic 
merit, plainly stands out as 

(A) an anomaly 

(B) an inaccuracy 

(C) an Inference 

(D) a misnomer 

(E) 8 pattern 



BASE N 

1,100 


OMIT 

3 


A * 

628 


B 

110 


C 

140 


D 

176 


E 

43 


M-TOT 
13.0 


SCALE 

3i 3R 


AE 
11.9 


CRIT 

XS76 


ITEM # 

3 


M-0 
8.3 


M-A 

14.8 


M-B 

10.6 


M-C 
10.0 


M-D 
11.6 


M-E 

8.3 


p. TOT 
1.00 


P+ 
0.57 


AO 
12.3 


Rbis 
0.67 



Revised Item 

In the midst of so many evasive comments, this forthright statement, whatever its intrinsic 
merit, plainly stands out as . 

(A) an anomaly 

(B) an inaccuracy 

(C) a pfofiittdily 

(D) a misnomer 

(E) a paradigm 



BASE N 

1,045 


OMIT 

•2 


A * 

471 


B 

78 


c 

254 


D 

90 


E 

150 


M-TOT 
13.0 


SCALE 

3DGR 


AE 
13.5 


CRIT 

XS76 


ITEM # 

6 


M-0 
11.0 


M-A 

14.8 


M-B 

10.1 


M-C 

12.4 


M-D 

10.3 


M-E 

11.5 


p. TOT 
1.00 


P+ 

0.45 


AO 
13.5 


Rbis 

0.52 



ERIC 
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APPENDIX D: STATISTICAL RESULTS FOR ALL ITEMS (N=72) 



Objective 
MakeEASIER 



Item 
Type 

SNCP 
ANAL 
ANT 

AVG 



Avg Change 
in Delta 

-0.26 
-0.49 
-0.16 

-0.33 



Range 

-2.1 to 1.1 
-1.5 to 1.1 
-2.0 to 2.0 



Std Dev 

0.92 
. 0.96 
1.41 



MakeHARDER 



SNCP 
ANAL 
ANT 

AVG 



+030 
+ 1.29 
+0.95 

+0.7S 



-2.9 to 2.8 
+0.2 to 3.5 
-1.0 to 3.1 



1.32 
1.20 
0.99 



CONTROL 



AVG 



+0.12 







Avg Change 


Objective 


Fonn 


in Delta 


MakeEASIER 


XI 


+0.37 




X2 


-0.29 




X3 


-1.08 


MakeHARDER 


XI 


+0.35 




X2 


+1.38 




X3 


+0.82 



20 



Appendix D, continued 16 

DATA ON ALL EXPERIMENTAL ITEMS 





















ACC# 


FORM 


ITEM# 


TYPE 


HARD/EASY 


DELTA 


DELTA 


CHG 


RBIS 


V-099004 


Xl 


2 




Easier 


y.u 


in*; 


1 c 
L.J 


n iA 


V-076475 


Xi 


3 


CXT/^T> 


Easier 


Q A 

y.4 


Q Q 

o.y 


n *; 

-U.J 


n *;9 

U.JZ 


V-083010 


. Xi 


4 


CXT/^T> 


Easier 


Q A 

y.o 


1 n 7 

lU. / 


1 1 
1. 1 


n 49 
u.tz 


V-U7o4o2 


Xz 


1 




Easier 


Q n 
y.u 


8 1 
o. 1 


n Q 
-yj.y 


n 4S 


V-074935 


a2 


2 




Easier 


lU.O 


1 n 1 

lU. 1 


n *; 

-U.J 


n *;'i 

U.JJ 


V-085023 


vo 
X2 


3 


bNCr 


Easier 


1 r> Q 
lU.o 


Q Q 

y.y 


n Q 
-u.y 


n 91 

U.Zl 


X 7 r\0 1 O 1 O 

V-081813 


X3 


1 




Easier 


Q 1 

y. / 


8 1 
O.J 


1 4 
-1.4 


n 44 


V-U53534 


Xi 


2 




Easier 


Q Q 

y.o 


Q Q 

y.y 


n 1 

U. 1 


n *;q 
u.jy 


V-UoJOOl 


Xi 


i 




Easier 


1 n 7 


Q Q 

y.y 


n 8 

-U.O 


n S9 

U.JZ 


V-078014 


XI 


8 


ANAL 


Easier 


9.1 


10.1 


1.0 


0.37 


V-083088 


XI 


9 


ANAL 


Easier 


9.4 


8.5 


-0.9 


0.61 




V1 

X 1 


1 r> 
lU 


A XI A T 


Easier 


Q A 

y.o 




1 1 


n '^0 


V-Uoo/ /o 


"Vl 

XI 


1 0 


AXI AT 


Easier 


1 n 7 

lU. / 


118 
1 l.o 


1 1 


n ^^7 

U.J / 


V-Uol /4o 




Q 

o 


A XT A T 


Easier 


lU.U 


8 S 

O.J 


1 *; 
- i.j 


n 91 

U.^ 1 


v-uoyo/z 


XZ 


y 


A XT A T 


Easier 


1 n A 

lU.O 


Q A 

y.o 


1 n ■ 


n S'^ 

U.JJ 


\7 AOflCylyl 

v-uyyj44 


V7 
AZ 


1 r> 


AKT AT 
AJN AL 


Easier 


1 n A 

lU.O 


in*; 


n 1 


n 17 

U. 1 / 


V-Uo/ /4o 


Xz 


1 0 
IZ 


AM AT 
AJN AL 


Easier 


mo 


1 1 n 


n 1 


n 49 


\ 7 no 1 om 
V-UoloU/ 


Ai 


Q 
O 


AXI AT 
AIN AJL 


Easier 




7 1 


9 1 


n S'^ 

U.JJ 


v-uyo3 /y 


Ai 


Q 

y 


AM AT 

AIN AJL 


Easier 


in A 

lU.O 


1 n A 


n n 


n S'^ 

U.JJ 


V -uyo43y 


A3 


1 n 


AM AT 
AIN AL 


Easier 


10 7 


inn 


n 7 


n 49 


V -UojUO / 


VI 
A 1 


Zo 


ANT 


„ • 
l^asier 


9 S 


10 4 


0 9 


0 07 


V -UojUZU 


VI 
A 1 




AMT 
AIN 1 




in 1 


1 1 8 

1 1 .o 


1 7 


0 57 


v-uyjj3U 


Y1 
AI 




AMT 
AIN 1 


Easier 


inn 


1 n 4 






V-089430 


XI 


32 


ANT 


Easier 


10.5 


9.3 


-1.2 


0.58 


V-081310 


X2 


28 


ANT 


Easier 


9.4 


11.4 


2.0 


0.25 


V-097401 


X2 


29 


ANT 


Easier 


10.0 


11.5 


1.5 


0.49 


V-056614 


X2 


30 


ANT 


Easier 


10.1 


10.7 


0.6 


0.46 


V-076500 


X2 


32 


ANT 


Easier 


10.5 


10.8 


0.3 


0.64 


V-087868 


X2 


33 


ANT 


Easier 


10.7 


8.9 


-1.8 


0.70 


V-096399 


X3 


28 


ANT 


Easier 


9.8 


8.1 


-1.7 


0.40 


V-079173 


X3 


29 


ANT 


Easier 


10.1 


8.1 


-2.0 


0.51 


V-083572 


X3 


30 


ANT 


Easier 


10.0 


8.1 


-1.9 


0.29 


V-087499 


X3 


31 


ANT 


Easier 


10.9 


10.1 


-0.8 


0.48 



*DIF B item **DIF C item 
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ORIG 


NEW 




NEW 


ACC# 


FORM 


ITEM# 


TYPE 


HARD/EASY 


DELTA 


DELTA 


CHG 


RBIS 


V-074937 


XI 


5 


SNCP 


Harder 


11.0 


11.2 


0.2 


0.45 


V-095557 


XI 


6 


SNCP 


Harder 


11.0 


10.8 


-0.2 


0.51 


V-086165 


XI 


7 


SNCP 


Harder 


11.2 


11.4 


0.2 


0.44 


V.n87Q09 

V '\JO 1 Z7\}£^ 




A 




ndruer 


1 1 1 


1 1 


u.z 




V-099392 


X2 


5 


SNCP 


Harder 


11.2 


8.3 


-2.9 


0.39 


V-082513 


X2 


6 


SNCP 


Harder 


11.3 


14.1 


2.8 


0.51 


V-067413 


X2 


7 


SNCP 


Harder 


12.1 


12.6. 


0.5 


0.12 


V-077947 


X3 


4 


SNCP 


Harder 


11.0 


11.8 


0.8 


0.35 


V-076638 


X3 


5 


SNCP 


Harder 


11.1 


11.6 


0.5 


0.49 


V-099427 


X3 


6 


SNCP 


Harder 


11.9 


13.5 


1.6 


0.52 


V-087504 


X3 


7 


SNCP 


Harder 


13.1 


12.7 


-0.4 


0.29 


V-058701 


XI 


13 


ANAL 


Harder 


12.1 


12.3 


0.2 


0.26 


V-099419 


XI 


14 


ANAL 


Harder 


12.3 


12.8 


0.5 


0.55 


V-086169 


XI 


15 


ANAL 


Harder 


•12.4 


\2A 


0.0 


0.21 


V-083094 


XI 


16 


ANAL 


Harder 


12.6 


13.6 


1.0 


0.37 


V-092972 


X2 


13 


ANAT 


rial UCl 


19 1 


1 "x n 


fl Q 
\j.y 




V-065697 


X2 


14 


ANAL 


Harder 


12.2 


13.6 


1.4 


0.30 


V-078012 


X2 


15 


ANAL 


Harder 


13.1 


16.5 


3.4 


0.07 


V-093062 


X2 


16 


ANAL 


Harder 


13.4 


16.3 


2.9 


0.20 


V-099492 


X3 


13 


ANAL 


Harder 


12.0 


12.9 


0.9 


0.35 


V-083650 


X3 


14 


ANAL 


Harder 


12.1 


12.6 


0.5 


0.50 


V-079197 


X3 


15 


ANAL 


Harder 


12.3 


15.8 


3.5 


0.29 


V-096453 


X3 


16 


ANAL 


Harder 


13.1 


13.4 


0.3 


0.33 


V-099011 


XI 


34 


ANT 


Harder 


12.0 


12.9 


0.9 


0.62 


V-083721 


XI 


35 


ANT 


Harder 


12.1 


12.3 


0.2 


0.47 


V-079231 


XI 


36 


ANT 


Harder 


12.8 


14.3 


1.5 


0.43 


V-093081 


XI 


37 


ANT 


Harder 


13.3 


13.4 


0.1 


0.52 


V-092619 


XI 


38 


ANT 


Harder 


13.4 


13.5 


0.1 


0.36 


V-094672 


X2 


34 


ANT 


Harder 


12.0 


15.1 


3.1 


0.21 


V-092815 


X2 


35 


ANT 


Harder 


12.1 


13.4 


1.3 


0.23 


V-097403 


X2 


36 


ANT 


Harder 


12.8 


15.0 


2.2 


0.23 


V-083069 


X2 


37 


ANT 


Harder 


12.9 


14.5 


1.6 


0.30 


V-087508 


X2 


38 


ANT 


Harder 


13.1 


14.1 


1.0 


0.55 


V-098409 


X3 


33 


ANT 


Harder 


12.2 


11.8 


-0.4 


0.48 


V-087871 


X3 


34 


ANT 


Harder 


12.3 


13.0 


0.7 


0.47 


V-()42924 


X3 


35 


ANT 


Harder 


1.3.0 


13.9 


0.9 


0.35 


V-096313 


X3 


36 


ANT 


Harder 


13.3 


14.3 


1.0 


0.43 


V-092606 


X3 


37 


ANT 


Harder 


13.4 


15.4 


2.0 


-0.04 


V-098334 


X3 


38 


ANT 


Harder 


13.6 


12.6 


-1.0 


0.20 



*DIF B item **DIF C item 

22 
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ORIG 


NJbW 




WHW 


ACC# 


FORM 


ITEM# 


TYPE 


HARD/EASY 


DELTA 


DULIA 






V-098371 


XI 


11 


ANAL 


Control 


v.o 


V.O 


U.U 


U.41 


V-074995 


X2 


11 


ANAL 


Control 


10.6 


1 1 

11.0 


0.4 


U.4i 


V-099543 


X3 


12 


ANAL 


Control 


11.8 


1 o o 

12.3 


0.5 


U.3 / 


V-075040 


XI 


33 


ANT 


Control 


10.6 


1 n O 

lU.V 


0.3 


U.5o 


V-099563 


XI 


17 


RCMP 


Control 


ll.U 


1U.2 


-O.O 


U.OU 


V-099564 


XI 


18 


RCMP 


Control 


13.5 


13.5 


0.0 


U.iV 


V-099565 


XI 


19 


RCMP 


Control 


13.6 


13.5 


n 1 
-0.1 


U.4o 


V-099566 


XI 


20 


RCMP 


Control 


14.0 


13.U 


1 n 
-l.U 


n /in 

U.4U 


V-087794 


XI 


21 


RCMP 


Control 


lO.O 


■\C\ A 

1U.4 


-U.Z 


U.3D 


V-0877% 


XI 


22 


RCNIP 


Control 


9.5 


V. / 


U.Z 


n /iQ 
U.hV 


V-087800 


XI 


23 


RCMP 


Control 


12.3 


12.1 


-O.Z 


U.44 


V-087799 


XI 


24 


RCMP 


Control 


11.3 


lU. / 


-0.6 


U.5o 


V-087802 


XI 


25 


RCMP 


Control 


13.5 


1 y4 1 

14.1 


U.O 


n /1C 
U.4D 


V-087795 


XI 


26 


RCMP 


Control 


11^ 

11.6 


1 1 o 
11.5 


O.Z 


n CO 
U.Do 


BE000616 


XI 


27 


RCMP 


Control 


11 A 

11.4 


1 1 o 

11.2 


-O.Z 


n io 


IF000114 


X2 


18 


RCMP 


Control 


14.2 


14.3 


0.1 


U.3/ 


IF000115 


X2 


19 


RCMP 


Control 


1 O O 

13.2 


1 O A 

13.0 


-0.2 


n ylC 

U.45 


IF000119 


X2 


21 


RCMP 


Control 


14.3 


14.9 


U.O 


U.iU 


IF000133 


X2 


22 


RCMP 


Control 


13./ 


13. / 


U.U 




V-093692 


X2 


25 


RCMP 


Control 


14.3 


13.8 


-0.5 


0.43 


V-093693 


X2 


26 


RCMP 


Control 


14.4 


14.2 


-0.2 


0.33 


V-093694 


X2 


27 


RCMP 


Control 


12.8 


13.1 


0.3 


0.39 


HT000415 


X3 


24 


RCMP 


Control 


8.1 


9.1 


1.0 


0.47 


HT000417 


X3 


25 


RCMP 


Control 


11.7 


12.7 


1.0 


0.44 


HT000438 


X3 


26 


RCMP 


Control 


8.2 


9.3 


1.1 


0.41 


HT000418 


X3 


27 


RCMP 


Control 


10.1 


10.9 


0.8 


0.15 



*DIF B item **DIF C item 
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APPENDIX E: STATISTICAL RESULTS FOR ACCEPTABLE (RBIS>.20) 

ITEMS (//=67) 



Objective 
MakeEASIER 



Item 

SNCP 
ANAL 
ANT 
ANAL + ANT 

AVG 



Avg Change 
in Delta 

-0.26 
-0.53 
-0.25 
-0.38 

-0.33 



Range 

-1.4 to 1.5 
-2.1 to 1.1 
-2.0 to 2.0 



Std Dcv 

0.92 
1.00 
1.44 



MakeHARDER 



SNCP 
ANAL 
ANT 
ANAL + ANT 

AVG 



+0.28 
+ 1.10 
+0.88 
+0.97 

+0.78 



-2.9 to 1.6 
+0.2 to 3.5 
-1.0 to 3.1 



1.39 
1.07 
0.99 



CONTROL 



AVG 



+0.12 



Distribution of Intended Changes in Delta 



AChange 0.3-0.7 0.8-1.2 13-1.7 1.8-2.2 23-Z7 2.8-3.2 33-3.7 
# of items 8 15 9 5 0 3 1 



24 



Appendix E, continued 20 

DATA ON ACCEPTABLE EXPERIMENTAL ITEMS 















NEW 




NEW 


ACC# 


FORM 


rrEM# 


TYPE 


HARD/EASY 


DELTA 


DELTA 


CHG 


RBIS 


v-uyyuu4 


V1 








9 0 


10.5 


1.5 


0.34 


V-U/t>4 IJ 


VI 


■X 






9 4 


8.9 


-0.5 


0.52 


V-Uo3UlU 


Y1 








9 6 


10.7 


1.1 


0.42 


V-U ItyvSz. 


AZ 


1 
1 






9.0 


8.1 


-0.9 


0.45 


V-U /4yjj 


AZ 








10.6 


10.1 


-0.5 


0.53 


V -Uo JUZ J 


Y? 
AZ 


•5 
J 






10.8 


9.9 


-0.9 


0.21 


V-Uolol J 


Aj 


1 

1 






9.7 


8.3 


-1.4 


0.44 




Y1 
Aj 








9.8 


9.9 


0.1 


0.59 


V-UODool 


Y'J 
Aj 


•5 






10.7 


9.9 


-0.8 


0.52 


V-078014 


XI 


8 


ANAL 


Easier 


9.1 


10.1 


1.0 


0.37 


V-Uo:)Uoo 


Y1 
Al 


Q 


AMAT 




9 4 


, 8.5 


-0.9 


0.61 


\/ nQoi /io 
V-UoV 10/ 


Y1 
Al 


in 

lU 


ANAT 




9 6 


8.3 


-1.3 


0.39 


V-UOO/ /o 


Y1 
Al 


19 

iz 


ANAT 




10.7 


11.8 


1.1 


0.37 


V -Uo 1 / 'to 


Y9 
AZ 


o 
o 


ANAT 




10.0 


8.5 


-1.5 


0.21 


V-UOVO/ii 


Y9 
AZ 


Q 


ANAT 




10.6 


9.6 


-1.0 


0.53 


V -Uo / /Ho 


Y9 
AZ 


19 


ANAT 




10.9 


11.0 


0.1 


0.42 


V -UO 1 ou / 




8 

o 


ANAL 


Pasier 


9.4 


7.3 


-2.1 


0.53 


V '\j^/yjD 1 " 


X3 


Q 


ANAL 


Easier 


10.6 


10.6 


0.0 


0.53 


V U-'Vt J" 


X3 


10 


ANAL 


Easier 


10.7 


10.0 


-0.7 


0.42 


V -UOJU-6U 


Y1 

1 


30 


ANT 


Easier 


10.1 


11.8 


1.7 


0.57 


V -U7 J J JU 




31 


ANT 


Easier 


10.0 


10.4 


0.4 


0.55 


V-089430 


XI 


32 


ANT 


Easier 


10.5 


9.3 


-1.2 


0.58 


V-081310 


X2 


28 


ANT 


Easier 


9.4 


11.4 


• 2.0 


0.25 


V-097401 


X2 


29 


ANT 


Easier 


10.0 


11.5 


1.5 


0.49 


V-056614 


X2 


30 


ANT 


Easier 


10.1 


10.7 


0.6 


0.46 


V-076500 


X2 


32 


ANT 


Easier 


10.5 


10.8 


0.3 


0.64 


V-087868 


X2 


33 


ANT 


Easier 


10.7 


8.9 


-1.8 


0.70 


V-096399 


X3 


28 


ANT 


Easier 


9.8 


8.1 


-1.7 


0.40 


V-079173 


X3 


29 


ANT 


Easier 


10.1 


8.1 


-2.0 


0.51 


V-083572 


X3 


30 


ANT 


Easier 


10.0 


8.1 


-1.9 


0.29 


V-087499 


X3 


31 


ANT 


Easier 


10.9 


10.1 


-0.8 


0.48 



*DIF B item **DIF C item 
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Appendix E, continued 
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ORIG 


NEW 




NEW 


ACC# 


FORM 


ITEM# 


TYPE 


HARD/EASY 


DELTA 


DELTA 


CHG 


RBIS 


V-074937 


XI 


5 


SNCP 


Harder 


11.0 


11.2 


0.2 


0.45 


V-095557 


XI 


6 


SNCP 


Harder 


11.0 


10.8 


-0.2 


0.51 


V-086165 


XI 


7 


SNCP 


Harder 


11.2 


11.4 


0.2 


0.44 


V-087902 


X2 


4 


SNCP 


Harder 


11.1 


11.3 


0.2 


0.54 


V-099392 


X2 


5 


SNCP 


Harder 


11.2 


8.3 


-2.9 


0.39 


V-082513 


X2 


6 


SNCP 


Harder 


11.3 


14.1 


2.8 


0.51 


V-077947 


X3 


4 


SNCP 


Harder 


11.0 


11.8 


0.8 


0.35 


V-076638 


X3 


5 


SNCP 


Harder 


11.1 


11.6 


0.5 


0.49 


V-099427 


X3 


6 


SNCP 


Harder 


11.9 


13.5 


1.6 


0.52 


V-087504 


X3 


7 


SNCP 


Harder 


13.1 


12.7 


-0.4 


0.29 


V-058701 


XI 


13 


ANAL 


Harder 


12.1 


12.3 


0.2 


0.26 


V-099419 


XI 


14 


ANAL 


Harder 


12.3 


12.8 


0.5 


0.55 


V-086169 


XI 


15 


ANAL 


Harder 


12.4 


12.4 


0.0 


0.21 


V-083094 


XI 


16 


ANAL 


Harder 


12.6 


13.6 


1.0 


0.37 


V-092972 


X2 


13 


ANAL 


Harder 


12.1 


13.0 


0.9 


0.29 


V-065697 


X2 


14 


ANAL 


Harder 


12.2 


1.3.6 


1.4 


0.30 


V-093062 


X2 


16 


ANAL 


Harder 


13.4 


16.3 


2.9 


0.20 


V-099492 


X3 


13 


ANAL 


Harder 


12.0 


12.9 


0.9 


0.35 


V-083650 


X3 


14 


ANAL 


Harder 


12.1 


12.6 


0.5 


0.50 


V-079197 


X3 


15 


ANAL 


Harder 


12.3 


15.8 


3.5 


0.29 


V-096453 


X3 


16 


ANAL 


Harder 


13.1 


13.4 


0.3 


0.33 


V-099011 


XI 


34 


ANT 


Harder 


12.0 


12.9 


0.9 


0.62 


V-083721 


XI 


35 


ANT 


Harder 


12.1 


12.3 


0.2 


0.47 


V-079231 


XI 


36 


ANT 


Harder 


12.8 


14.3 


1.5 


0.43 


V-093081 


XI 


37 


ANT 


Harder 


13.3 


13.4 • 


0.1 


0.52 


V-092619 


XI 


38 


ANT 


Harder 


13.4 


13.5 


0.1 


0.36 


V-094672 


X2 


34 


ANT 


Harder 


12.0 


15.1 


3.1 


0.21 


V-092815 


X2 


35 


ANT 


Harder 


12.1 


13.4 


1.3 


0.23 


V-097403 


X2 


36 


ANT 


Harder 


12.8 


15.0 


2.2 


0.23 


V-083069 


X2 


37 


ANT 


Harder 


12.9 


14.5 


1.6 


0.30 


V-087508 


X2 


38 


ANT 


Harder 


13.1 


14.1 


1.0 


0.55 


V-098409 


X3 


33 


ANT 


Harder 


12.2 


11.8 


-0.4 


0.48 


V-087871 


X3 


34 


ANT 


Harder 


12.3 


13.0 


0.7 


0.47 


V-042924 


X3 


35 


ANT 


Harder 


13.0 


13.9 


0.9 


0.35 


V-0%313 


X3 


36 


ANT 


Harder 


13.3 


14.3 


1.0 


0.43 


V-098334 


X3 


38 


ANT 


Harder 


13.6 


12.6 


-1.0 


0.20 



*DIF B item **DIF C item 
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APPENTDIX F: FREQUENCY DISTRIBUTION FOR HARD 
EXPERIMENTAI. ANALOGIES AND ANTONYMS WITH ACCEPTABLE RBIS 



# Produced Freq/ # Produced Freq/ 

Delta Range in Exp.Pretsts Pretst* with Good DIF Pretst* 



<n.o 


0 


0.0 


0 


0,0 


11.0-11.9 


1 


0.7 


0 


0.0 


IZO - 1Z9 


8 


5.7 


6 


4.3 


13.0 - 13.9 


9 


6.4 


7 


5.0 


14.0 - 14.9 


4 


2.9 


4 


2.9 


15.0 - 15.9 


3 


2.1 


3 


2.1 


>16.0 


1 


0.7 


1 


0.7 


Unacceptable Rbis 


2 


1.4 


2 


1.4 


TOTAL 


28 




23 





*Frcqucncy/pretest calculated by dividing the number of items produced in each delta range by 
28/20 ([total # hard ANALS & ANTS in exp. pretests]/[# ANALS & ANTS per 1 pretest]). This 
gives the frequency to be expected were an entire pretest's worth of analogies and antonyms 
revised to be made harder according to the experimental procedure. 
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APPENDIX G: CUMULATIVE GRE PRETEST STATISTICS 
(K-3KGR3m^ K-3KGR3m3»^^, and K-3KGR1«a^) 

Tally #1 

Number of items with rbi,> .25, by item type and delta 





<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 


TOT 


RCMP 


2 


8 


15 


30 


46 


32 


21 


10 


1 


0 


165 


SNCP 


10 


10 


14 


13 


9 


5 


5 


7 


1 


1 


75 


ANAL 


4 


3 


11 


9 


19 


11 


10 


5 


8 


1 


81 


ANTM 


3 


9 


9 


13 


17 


20 


14 


17 


4 


4 


110 


TOTAL 


19 


■30 


49 


65 


91 


68 


50 


39 


14 


6 


431 


YIELD 


1.7 


2.7 


4.5 


5.9 


8.3 


6.2 


4.5 


3.5 


1.3 


0.5 


38.9 



Tally #2 

Number of items with T^i>. .25 and acceptable DIF, by item type and delta 





<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 


TOT 


RCMP 


2 


8 


10 


28 


38 


30 


13 


9 


1 


0 


140 


SNCP 


9 


6 


11 


9 


7 


3 


4 


7 


1 


1 


60 


ANAL 


3 


2 


3 


5 


12 


5 


8 


5 


8 


1 


52 


ANTM 


2 


4 


4 


8 


8 


17 


9 


16 


4 


4 


80 


TOTAL 


16 


20 


28 


50 


65 


55 


34 


37 


14 


6 


332 


YIELD 


1.5 


1.8 


2.5 


4.5 


5.9 


5.0 


3.1 


3.4 


1.3 


0.5 


29.5 



Tally #3 

Number of items with t^„< .25, by item type 
RCMP: 15 SNCP: 2 ANAL: 10 ANTM: 11 TOTAL: 38 
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FREQUENCY DISTRTOUTION FOR STANDARD 
ANALOGIES AND ANTONYMS 



# Produced Freq/ # Produced Freq/ 



Delta Range 


in Pretsts 


Pretst* 


with Good DBF 


Pretst* 


<8.0 


7 


0.6 


5 


0.5 


8.0 - 8.9 


12 


1.1 


6 


0.5 


9.0 - 9.9 


20 


1.8 


7 


0.6 


10.0 - 10.9 


22 


2.0 


13 


1.2 


11.0 - 11.9 


36 


3.3 


20 


1.8 


IZO - 1Z9 


31 


2.8 


22 


2.0 


13.0 - 13.9 


24 


2.2 


17 


1.5 


14.0 - 14.9 


22 


2.0 


21 


1.9 


15.0 - 15.9 


12 


1.1 


12 


1.1 


>16.0 


5 


0.5 


5 


0.5 



*Frequency/pretest calculated by dividing the number of items produced in each delta range by 1 1 
[# of pretests' worth of data the raw figures are based on]. 
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APPENDIX H: VALUES FOR GRE PRETEST ITEMS BY DELTA 
(MARGINAL UTILITY ANALYSIS) 



Delta: 


<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 






Avg # used in 


























4 Final Fonns: 


20 


24 


36 


28 


32 


44 


44 


44 


23 


9 






Avg yield/ 


























Std-Pretest: 


1.5 


1.8 


2.5 


4.5 


5.9 


5.0 


3.1 


3.4 


1.3 


0.5 






(All item types)* 
















































Item 


Cost/ 
























Used 


Item 


Pretest Yields 


<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 


Yield" 


Used*" 


1 Pretest 


1 


2 


3 


5 


6 


5 


3 


3 


1 


1 


30 


677 


2 Pretests 


3 


4 


5 


9 


12 


10 


6 


7 


3 


1 


30 


677 


3 Pretests 


4 


5 


8 


14 


18 


15 


9 


10 


4 


2 


30 


677 


4 Pretests 


6 


7 


10 


18 


24 


20 


12 


13 


5 


2 


30 


677 


5 Pretests 


7 


9 


13 


23 


30 


25 


15 


17 


6 


3 


30 


677 


6 Pretests 


9 


11 


15 


21 


35 


30 


19 


20 


8 


3 


30 


677 


7 Pretests 


10 


13 


18 


32 




35 


22 


24 


9 


4 


24 


846 


8 Pretests 


12 


15 


20 






40 


25 


27 


10 


4 


19 


1,048 


9 Pretests 


13 


lo 


15 








/o 




1 1 
1 1 


c 

J 


1Q 




10 Pretests 


15 


18 


25 






50 


31 


34 


13 


5 


19 


1,048 


i I Pretests 


16 


20 


28 








34 


37 


14 


6 


14 


1,419 


12 Pretests 


17 


22 


31 








37 


40 


15 


7 


14 


1,419 


13 Pretests 


19 


24 


33 








40 


44 


17 


7 


14 


1,419 


14 Pretests 


20 


25 


36 








43 


47 


18 


8 


14 


1,419 


1j t retests 




LI 


DO 












1Q 


O 

o 






16 Pretests 






41 








49 




20 


9 


7 


2,683 


17 Pretests 


















22 


9 


2 


11,000 


18 Pretests 


















23 


10 


2 


11,000 


19 Pretests 


















24 


10 


2 


11,000 


20 Pretests 


















25 


11 


2 


11,000 


TOT S/!x:"' 


22,033 


27,541 


45,386 


22,308 


24,000 


40,253 


55,112 


50,950 


78,693 


33,726 






20 Pretests 


























Yield: 


29 


36 


51 


91 


118 


100 


62 


67 


25 


11 






AVG $/AItem: 


151 


151 


892 


245 


203 


403 


892 


757 


3,092 


3,092 







* Data from Appendix G, Tally #2 

** # of items in n'th pretest needed to meet target goals for four final forms 

*** Calculated by dividing $20,000 [cost/pretest] by (# of items used in n'th pretest) 

**** Calculated by summing for each A the # of items used in the n'th pretest times the cost/item used 
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APPENDIX I: STANDARD AND EXPERIMENTAL PRETEST YIELDS 

(ACCEPTABLE DIP) 



STANDARD PRETESTING YIELDS 



Delta: <8 8 9 10 11 12 13 14 15 >16 

Avg # used in 

4 Final Forms: 20 24 36 28 . 32 44 44 44 23 9 
Avg yield/ 

StdPretest: 1.5 1.8 2.5 4.5 5.9 5,0 3.1 3.4 1.3 0.5 
(All Item Types)* 



Standard 



Pretest Yields 


<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 


1 Pretest 


1 


2 


3 


5 


6 


5 


3 


3 


1 


1 


2 Pretests 


3 


4 


5 


9 


12 


10 


6 


7 


3 


1 


3 Pretests 


4 


5 


8 


14 


18 


15 


9 


10 


4 


2 


4 Pretests 


6 


7 


10 


18 


24 


20 


12 


13 


5 


2 


5 Pretests 


7 


9 


13 


23 


30 


25 


15 


17 


6 


3 


6 Pretests 


9 


11 


15 


27 


35 


30 


19 


20 


8 


3 


7 Pretests 


10 


13 


18 


32 


41 


35 


22 


24 


9 


4 


8 Pretests 


12 


15 


20 


36 


47 


40 


25 


27 


10 


4 


9 Pretests 


13 


16 


23 


41 


53 


45 


28 


30 


11 


5 


10 Pretests 


15 


18 


25 


45 


59 


50 


31 


34 


13 


5 


11 Pretests 


16 


20 


28 


50 


65 


55 


34 


37 


14 


6 


12 Pretests 


17 


22 


31 


55 


71 


60 


37 


40 


15 


7 


13 Pretests 


19 


24 


33 


59 


77 


65 


40 


44 


17 


7 


14 Pretests 


20 


25 


36 


64 


83 


70 


43 


47 


18 


8 


15 Pretests 


22 


27 


38 


68 


89 


75 


46 


50 


19 


8 


16 Pretests 


23 


29 


41 


73 


95 


80 


49 


54 


20 


9 


17 Pretests 


25 


31 


43 


77 


100 


85 


53 ■. 


57 


22 


9 


18 Pretests 


26 


33 


46 


82 


106 


90 


56 


61 


23 


10 


19 Pretests 


28 


35 


48 


86 


112 


95 


59 


64 


24 


10 


20 Pretests 


29 


36 


51 


91 


118 


100 


62 


67 


25 


11 



•See Appendix G, Tally #2 
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Appendix I, continued 27 

EXPERIMENTAL PRETESTING YIELDS 



Delta: 


<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16. 


Avg # used in 






















4 Hnal Forms: 


20 


24 


36 


28 


32 


44 


44 


44 


23 


9 


Avg yield/ 






















Exp.Pretest: 


1.0 


1.3 


1.9 


3.4 


4.1 


7.3 


6.5 


4.3 


2.3 


0.8 


(All Item Types) 






















Experimental 






















Pretest Yields 


<8 


8 


9 


10 


11 


12 


13 


14 


15 


>16 


1 Pretest 


1 


1 


2 


3 


4 


7 


7 


4 


2 


1 


2 Pretests 


2 


3 


4 


7 


8 


15 


13 


9 


5 


2 


3 Pretests 


3 


4 


6 


10 


12 


22 


20 


13 


7 


2 


4 Pretests 


4 


5 


8 


13 


16 


30 


27 


18 


10 


3 


5 Pretests 


5 


6 


10 


17 


20 


37 


34 


22 


12 


4 


6 Pretests 


6 


8 


11 


20 


25 


45 


40 


27 


14 


5 


7 Pretests 


7 


9 


13 


24 


29 


52 


47 


31 


17 


6 


8 Pretests 


8 


10 


15 


27 


33 


60 


54 


35 


19 


7 


9 Pretests 


9 


11 


17 


30 


37 


67 


61 


A f\ 

40 


22 


7 


10 Pretests 


10 


13 


19 


34 


41 


74 


67 


44 


24 


8 


11 Pretests 


11 


14 


21 


37 


45 


82 


74 


49 


26 


9 


12 Pretests 


12 


15 


23 


40 


49 


89 


81 


53 


29 


10 


13 Pretests 


13 


17 


25 


44 . 


53 


97 


87 


57 


31 


11 


14 Pretests 


14 


18 


27 


47 


57 


104 


94 


62 


34 


12 


15 Pretests 


15 


19 


29 


50 


61 


112 


101 


66 


36 


12 


16 Pretests 


16 


20 


31 


54 


65 


119 


108 


71 


38 


13 


17 Pretests 


17 


22 


32 


57 


70 


127 


114 


75 


41 


14 


18 Pretests 


18 


23 


34 


61 


74 


134 


121 


80 


43 


15 


19 Pretests 


19 


24 


36 


64 


78 


141 


128 


84 


46 


16 


20 Pretests 


20 


25 


38 


67 


82 


149 


135 


88 


48 


17 



♦♦Aggregation of data from Appendix F and Appendix G, Tally #2 for RCMP and SNCP 
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Appendix I, continued 



28 



COST TO ATTAIN REQUISITE NUMBER OF ITEMS 
FOR FOUR FINAL FORMS 



N, = # of standard pretests 

Nj = # of experimental pretests 

Vj = $value of experimental pretest items 

Cost = (N. X $2a000[$/std. pretest]) + (N, x $19,150[$/exp. pretest]) + (N, x V.) 

Pretested itenas: V, = $0 V, = $10,985 

0 Exp.Prtst + 20 Std.Prtst = $400,000 

1 Exp.Prtst + 18 StdPrtst = $379,150 $390,135 

2 Exp.Prtst + 16 StdPrtst = $358,300 $380,269 

3 Exp.Prtst + 14 Std.Prtst = $337,450 $370,404 

4 Exp.Prtst + 13 Std.Prtst = $336,600 $380,538 

5 Exp.Prtst + 12 Std.Prtst = $335,750 $390,673 

6 Exp.Prtst + 11 Std.Prtst = $334,900 $400,807 
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